Cognitive disorder human-computer interaction method and system based on emotion monitoring

ABSTRACT

Disclosed are a cognitive disorder human-computer interaction method and system based on emotion monitoring. The human-computer interaction method includes the following steps: obtaining a cognitive ability level of a user; generating a human-computer interaction scheme according to the cognitive ability level; generating an N th  human-computer interaction task according to the human-computer interaction scheme; carrying out emotion monitoring on the performance of the user in the N th  human-computer interaction task, and returning to the previous step of generating an N th  human-computer interaction task according to the human-computer interaction scheme, whereby N=N+1 until all tasks in the human-computer interaction scheme are completed; and carrying out, in the process of human-computer interaction, a relaxed human-computer interaction between two adjacent human-computer interaction tasks, and selecting a task type of the relaxed human-computer interaction according to an emotion comprehensive index of the user in the previous human-computer interaction task.

BACKGROUND Technical Field

The present disclosure relates to a cognitive disorder human-computerinteraction method based on emotion monitoring, and also relates to acorresponding cognitive disorder human-computer interaction system,which fall within the technical field of human-computer interaction.

Related Art

Cognition refers to an ability of the human brain to change intocognition and learned knowledge by processing inner activities aftercontacting with external information or things, that is, to learn newthings, acquire knowledge and apply the learned knowledge. Cognitivefunctions include memory, comprehension, computing power, languageabilities, visual space abilities, judgment and comprehension abilities,etc. If one or more of the above-mentioned cognitive functions areimpaired, the impairment may be considered as a cognitive dysfunction.The cognitive dysfunction not only affects living abilities, but alsoaffects social abilities.

According to statistics, 10%-20% of the elderly over 65 years old havemild cognitive impairment. The study found that about 10% of thepatients with mild cognitive impairment are deteriorated to cognitivedisorders after 1 year. The cognitive disorders are also known as seniledementia, brain degeneration, and dementia. The cognitive disorders havesymptoms such as memory decline, decreased thinking ability, influenceon communication with family, and frequent emotion instability. Thepatients with serious cognitive disorders will be disabled in self-care.

The cognitive dysfunction, if not prevented, will go through the processfrom mild cognitive impairment to cognitive disorder, and cognitivetraining is an effective way to improve brain cognitive dysfunction, andis one of the commonly used cognitive rehabilitation treatment means. Inaddition to the traditional artificial cognitive training, thecombination of computer technology with cognitive rehabilitationtraining can effectively improve the efficiency of cognitive training,and training data results can be managed in a unified manner, therebyfacilitating view and analysis.

However, in the existing cognitive disorder evaluation schemes, userscannot evaluate or intervene in the influence of real-time emotion statefluctuations of users on training effects in the process of evaluationand training.

SUMMARY

The primary technical problem to be solved by the present disclosure isto provide a cognitive disorder human-computer interaction method basedon emotion monitoring, so as to improve the effect of cognitive disorderhuman-computer interaction.

Another technical problem to be solved by the present disclosure is toprovide a cognitive disorder human-computer interaction system based onemotion monitoring.

To achieve the foregoing technical objectives, the present disclosureuses the following technical solutions:

According to a first aspect of embodiments of the present disclosure, acognitive disorder human-computer interaction method based on emotionmonitoring is provided, including the following steps:

-   obtaining a cognitive ability level of a user;-   generating a human-computer interaction scheme according to the    cognitive ability level;-   generating an N^(th) human-computer interaction task according to    the human-computer interaction scheme, N being a positive integer;    and-   carrying out emotion monitoring on the performance of the user in    the N^(th) human-computer interaction task, and returning to the    previous step of generating an N^(th) human-computer interaction    task according to the human-computer interaction scheme, whereby    N=N+1 until all tasks in the human-computer interaction scheme are    completed.

Preferably, in the process of human-computer interaction, a relaxedhuman-computer interaction is carried out between two adjacenthuman-computer interaction tasks, and a task type of the relaxedhuman-computer interaction is selected according to an emotioncomprehensive index of the user in the previous human-computerinteraction task.

When the emotion comprehensive index is positive, a human-computerinteraction task of a high strength challenge class is selected, whenthe emotion comprehensive index is negative, a human-computerinteraction task of a high relaxation and stress relief class isselected, and when the emotion comprehensive index is neutral, ahuman-computer interaction task is randomly selected.

Preferably, when the user completes the relaxed human-computerinteraction, the next human-computer interaction task is pushedaccording to a human-computer interaction result of the previoushuman-computer interaction task and the emotion comprehensive index ofthe user after the relaxed human-computer interaction.

Preferably, the human-computer interaction result is divided into good,moderate and poor, and the emotion comprehensive index is divided intopositive, normal and negative.

When the human-computer interaction result is good, task pushing issuccessively carried out according to upgrade, upgrade, and reducepushing and upgrade in the order of the emotion comprehensive indexbeing positive, normal and negative.

When the human-computer interaction result is moderate, task pushing issuccessively carried out according to increase pushing, maintain, andreduce pushing in the order of the emotion comprehensive index beingpositive, normal and negative.

When the human-computer interaction result is poor, task pushing issuccessively carried out according to increase pushing and degrade,degrade, and degrade in the order of the emotion comprehensive indexbeing positive, normal and negative.

Preferably, acquiring an emotion comprehensive index of a userspecifically includes:

-   acquiring expression information of the user in each collection time    period;-   acquiring expression features in the expression information, and    comparing the expression features with an Asian face database so as    to obtain proportions of a positive emotion, a negative emotion and    a neutral emotion in the expression information, the emotion with    the maximum proportion being a current emotion of the user;-   determining proportions of positive emotions, negative emotions and    neutral emotions in all the current emotions of the user in the    whole process of human-computer interaction;-   acquiring a current emotion state of the user based on the    proportions of the positive emotions, the negative emotions and the    neutral emotions in all the current emotions of the user;-   acquiring an emotion change state of the user based on an emotion    proportion and a time change in each collection time period;-   acquiring an emotion fluctuation state of the user based on the    emotion proportion and the time change in each collection time    period; and-   obtaining the emotion comprehensive index of the user according to    at least two of the current emotion state, the emotion change state    and the emotion fluctuation state.

Preferably, the acquiring a current emotion state of the user based onthe proportions of the positive emotions, the negative emotions and theneutral emotions in all the current emotions of the user specificallyincludes:

-   when the proportion of the positive emotion of the user is greater    than or equal to a first threshold, determining the current emotion    state of the user as a positive state;-   when the proportion of the negative emotion of the user is greater    than or equal to a second threshold, determining the current emotion    state of the user as a negative state; and-   when both the proportions of the positive emotion and the negative    emotion of the user are less than the second threshold and the    proportion of the neutral emotion of the user is greater than or    equal to a third threshold, determining the current emotion state of    the user as a neutral state.

Preferably, the acquiring an emotion change state of the user based onan emotion proportion and a time change in each collection time periodspecifically includes:

-   taking the time change as an X value, taking the emotion proportion    of the user in each collection time period as a Y value, and    performing regression analysis using the X value and the Y value to    obtain a slope;-   acquiring a slope of the positive emotion and the negative emotion    changing with time throughout the full-time history;-   obtaining an emotion with the maximum slope change by comparison;-   if the emotion with the maximum slope change is a descending    negative emotion or an ascending positive emotion and the slope    change is significantly not equal to 0, determining the emotion    change state of the user as a positive change;-   if the emotion with the maximum slope change is an ascending    negative emotion or a descending positive emotion and the slope    change is significantly different from 0, determining the emotion    change state of the user as a negative change; and-   if the maximum slope change is not significantly distinguished from    0, determining the emotion change state of the user as no change.

Preferably, the acquiring an emotion fluctuation state of the user basedon the emotion proportion and the time change in each collection timeperiod specifically includes:

-   defining the emotion with the highest proportion in each collection    time period as the current principal emotion component;-   comparing the principal emotion components of two adjacent    collection time periods, and analyzing whether the principal emotion    component changes;-   if the principal emotion component changes, recording as 1,    otherwise, recording as 0;-   calculating a principal emotion component change ratio in each task    stage according to the record results of whether the principal    emotion component changes; and-   if the principal emotion component change ratio is less than a    fourth threshold, determining as a normal fluctuation, if the    principal emotion component change ratio is greater than or equal to    the fourth threshold and less than or equal to a fifth threshold,    determining as a slight fluctuation, and if the principal emotion    component change ratio is greater than the fifth threshold,    determining as a severe fluctuation.

User information and user informed consent are acquired, the userinformed consent including at least an evaluation content, ahuman-computer interaction content introduction, a user informationcollection range, and a user information usage range.

If the user agrees, cognitive evaluation is started and a facecollection device is initiated to start collecting user expression data,and if the user disagrees, the evaluation is stopped.

According to a second aspect of embodiments of the present disclosure, acognitive disorder human-computer interaction system based on emotionmonitoring is provided, including:

-   an interaction unit, configured to receive information input by a    user or output information to the user;-   a storage unit, configured to store a computer program; and-   a processing unit, configured to read the computer program to    execute the cognitive disorder human-computer interaction method.

The present disclosure has the following technical effects:

The present disclosure provides a cognitive disorder human-computerinteraction method and system based on emotion monitoring. Cognitiveevaluation and human-computer interaction tasks are mainly included, anda variety of basic emotion information of a user in the process ofexecuting tasks is monitored simultaneously in real time, and an emotioncomprehensive index of the user is obtained. Subsequent human-computerinteraction tasks and states of the user are adjusted through theemotion comprehensive index, so as to achieve the purpose of improvingthe cognitive disorder human-computer interaction effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of a cognitive disorderhuman-computer interaction system based on emotion monitoring accordingto an embodiment of the present disclosure.

FIG. 2 is a flowchart of a cognitive disorder human-computer interactionmethod based on emotion monitoring according to an embodiment of thepresent disclosure.

FIG. 3 is a flowchart of cognitive evaluation of a user.

FIG. 4 is a flowchart of acquiring an emotion comprehensive index of auser.

FIG. 5 is a qualitative flowchart of acquiring a current emotion state.

FIG. 6 is a qualitative flowchart of acquiring an emotion change state.

FIG. 7 is a qualitative flowchart of acquiring an emotion fluctuationstate.

FIG. 8 is a qualitative flowchart of acquiring an emotion comprehensiveindex.

FIG. 9 is a flowchart of another cognitive disorder human-computerinteraction method based on emotion monitoring according to anembodiment of the present disclosure.

FIG. 10 is a flowchart of yet another cognitive disorder human-computerinteraction method based on emotion monitoring according to anembodiment of the present disclosure.

FIG. 11 is a schematic structural diagram of another cognitive disorderhuman-computer interaction system based on emotion monitoring accordingto an embodiment of the present disclosure.

DETAILED DESCRIPTION

The technical content of the present disclosure is described in detailbelow with reference to the accompanying drawings and specificembodiments.

In the present disclosure, an open source emotion recognition package(real-time-emotion-detection-master) is used to capture expression datain real time by taking a period T (for example, 0.5 seconds) as a unit(with a variable value) after acquiring a face position of a user by abuilt-in or external face collection device, face features are analyzed,and six basic emotions: happiness, neutrality, sadness, anger, disgust,and surprise are obtained according to a comparison with an Asian facedatabase. Each emotion has the following meanings:

-   Happiness: a positive emotion state in which the user is interest in    human-computer interaction and the human-computer interaction is low    in difficulty.-   Neutrality: a calm and stable emotion state in which the user    focuses on human-computer interaction and the human-computer    interaction is moderate in difficulty.-   Sadness: a depressed emotion state in which the user is immersed in    the personal emotion and hardly enters into the human-computer    interaction, or is self-denied due to high difficulty in    human-computer interaction.-   Anger: a disgusting and anxious emotion state in which the user is    dissatisfied in human-computer interaction and the human-computer    interaction is high in difficulty.-   Disgust: an emotion state in which the user does not like the    content of human-computer interaction and wants to avoid and end the    human-computer interaction due to poor experience brought by the    human-computer interaction.-   Surprise: an emotion state in which the user goes into    human-computer interaction, but the content or feedback of    human-computer interaction is inconsistent with expectations, or the    user does not understand the content of human-computer interaction    and is slightly negative in the human-computer interaction scene.

In one embodiment of the present disclosure, the six different emotionsare classified, where happiness belongs to positive emotions, neutralitybelongs to neutral emotions, and sadness, anger, disgust, and surprisebelong to negative emotions.

FIG. 1 shows a cognitive disorder human-computer interaction systembased on emotion monitoring according to an embodiment of the presentdisclosure. The system includes: an evaluation module 1, an evaluationanalysis module 2, a human-computer interaction module 3, ahuman-computer interaction analysis module 4, an emotion recognitionmodule 5, an emotion analysis module 6, a report display module 7, and acentral processing unit 8.

The evaluation module 1 is connected to the central processing unit 8and is configured to invoke a corresponding evaluation topic by means ofhuman-computer interaction so as to carry out cognitive evaluation on auser. The evaluation and analysis module 2 is connected to the centralprocessing unit 8 and is configured to carry out cognitive analysis onthe user according to an answer result of the user so as to determine acognitive impairment situation of the user. The human-computerinteraction module 3 is connected to the central processing unit 8 andis configured to carry out human-computer interaction of cognitive taskson the user according to the cognitive impairment situation of the user.The human-computer interaction analysis module 4 is connected to thecentral processing unit 8 and is configured to analyze a human-computerinteraction situation of the user according to a human-computerinteraction result of the user so as to determine whether the cognitionof the user is improved. The emotion recognition module 5 is connectedto the central processing unit 8, and is configured to capture a faceexpression of the user in real time during the evaluation process andthe human-computer interaction process. The emotion analysis module 6 isconnected to the central processing unit 8 (or connected to the emotionrecognition module 5) and is configured to compare the captured userexpressions with a face database so as to determine the emotion categoryof the user (i.e. belonging to one of positive, neutral or negativeemotions). The report display module 7 is connected to the centralprocessing unit 8 and is configured to output information, for example,display information such as the result of the cognitive evaluation ofthe user, the result of the human-computer interaction of the user andthe emotion state of the user. The central processing unit 8 isconfigured to execute a cognitive disorder human-computer interactionmethod.

The cognitive disorder human-computer interaction method based onemotion monitoring provided by the present disclosure will be describedin detail below:

First Embodiment

FIG. 2 shows a cognitive disorder human-computer interaction methodbased on emotion monitoring according to this embodiment, whichspecifically includes the following steps:

S10: Obtain a cognitive ability level of a user

User information is acquired, and cognitive evaluation is carried out onthe user according to an acquired evaluation topic, so as to obtain anevaluation result of the user.

As shown in FIG. 3 , the operation specifically includes the followingsteps S101-S103:

S101: Acquire user information and user informed consent to realizelogin.

Specifically, the user needs to register an account in advance, andinputs personal information to obtain a login account, so as to log in acognitive disorder human-computer interaction system. The user informedconsent specifically includes: evaluation, a human-computer interactioncontent introduction, a user information collection range, a userinformation usage range, and other information. If the user agrees,cognitive evaluation is started and an emotion collection device isinitiated to start collecting user expression data, and if the userdisagrees, the evaluation is stopped.

It will be appreciated that this step may be omitted.

S102: Acquire data information of the user in an evaluation topic.

Specifically, when the user confirms to start the cognitive evaluation,the system invokes the corresponding evaluation topic from a database(for example: scales of different levels), the data information of theuser in the evaluation topic is acquired according to an answersituation of the user for subsequent analysis.

S103: Analyze a cognitive impairment situation of the user according tothe data information of the user in the evaluation topic, and finallyobtain an evaluation result of the user, so as to subsequently push acorresponding level of human-computer interaction scheme for the user.

S20: Collect face information of the user to extract emotioninformation, and acquire an emotion comprehensive index of the user.

As shown in FIG. 4 , the operation specifically includes the followingsteps S201-S207:

S201: Acquire expression information of the user in each collection timeperiod.

In this embodiment, the expression information of the user is collectedsimultaneously during the cognitive evaluation of the user.Specifically, an open source emotion recognition package(real-time-emotion-detection-master) may be used to capture expressiondata in real time by 0.5 seconds as a collection time period (with avariable value) after acquiring a face position of the user by abuilt-in or external face collection device.

S202: Acquire expression features in the expression information, andcompare the expression features with a face database so as to obtainsimilarity proportions of a positive emotion, a negative emotion and aneutral emotion in the expression information.

Specifically, in the form of a video stream, the software intercepts andcaptures the current expression information of each frame according tothe face collection device in real time, analyzes expression emotionfeatures using a cascade classifier based on Haar features, and comparesthe expression emotion features with six standard emotion modelsconstructed by the face database (for example, an Asian face database)to obtain a standard similarity proportion of each emotion. The emotionwith the maximum similarity proportion is a current emotion of the user.That is, the standard similarities of various emotions are calculatedwithin a prescribed unit time, and the emotion with the highestsimilarity serves as the current emotion of the user. And then anemotion type of the user is determined according to the basic emotion(i.e. belonging to one of positive emotions, negative emotions andneutral emotions). For example, a collection time period is defined as0.5 seconds, a picture transmission frame rate of a camera is 20frames/second, and the software analyzes 10 frames of face emotions ofthe user in real time within the collection time period as shown in thefollowing table, so as to obtain a happiness similarity of 0.65, aneutrality similarity of 0.19, a sadness similarity of 0.003, an angersimilarity of 0.09, a disgust similarity of 0.0002, and a surprisesimilarity of 0.01. Then the user expression captured by the collectionunit is happiness, and it is determined that the current emotion is apositive emotion.

The software analyzes 10 frames of face emotions of the user in realtime within a collection time period as shown in Table 1:

TABLE 1 Number of frames in unit Anger Disgust Sadness HappinessSurprise Neutrality 1 0.082701 0.000300849 0.003102166 0.7102865580.010221709 0.147617802 2 0.072682 0.000176214 0.00394929 0.7963571550.018063443 0.080297284 3 0.042726111 5.40339E-05 0.0012460920.747539997 0.003184185 0.166447029 4 0.082605027 0.0002641380.004705108 0.490251958 0.015535718 0.298862606 5 0.1186215360.000295937 0.003787976 0.403211385 0.013765618 0.352795094 60.144133478 0.0003543 0.003119486 0.450796932 0.025949353 0.273674965 70.083963729 0.000137187 0.003360974 0.630872965 0.023129571 0.1867638538 0.036378726 0.000149024 0.005776993 0.901797056 0.0120351740.031774726 9 0.106588304 0.000135205 0.004938024 0.7281923290.014183545 0.109377623 10 0.113372497 0.000243247 0.0035217220.613607347 0.008243959 0.205394134 Average of this unit 0.0883770.000211 0.003751 0.647291 0.014431 0.185301

It will be appreciated that because the similarity proportion of eachemotion is to compare the collected face expressions with the basicemotions individually, the face expressions of each user will have theproportions of positive emotions, negative emotions and neutralemotions, but the sum of proportions of positive emotions, negativeemotions and neutral emotions is not necessarily equal to 1. After thecomparative analysis, the emotion with the maximum similarity proportionis found as the current emotion of the user under the face expressions.

S203: Calculate proportions of positive emotions, negative emotions andneutral emotions of the user within a collection time.

Within the time range of a cognitive evaluation stage or ahuman-computer interaction stage, by determining all the currentemotions of the user within a collection time, the number of positiveemotions, the number of negative emotions and the number of neutralemotions of the user are obtained within the time period. Then, theproportions of positive emotions, negative emotions and neutral emotionsare calculated according to the total amount of the current emotionswithin the collection time. For example, in the cognitive evaluationstage, the user takes 5 minutes for cognitive evaluation, and then theface collection device obtains 600 current emotions of the user withinthe 5 minutes. If there are 300 positive emotions, 200 negative emotionsand 100 neutral emotions among the 600 current emotions, the proportionof the positive emotions is 50%, the proportion of the negative emotionsis 33.3%, and the proportion of the neutral emotions is 16.7%.

S204: Acquire a current emotion state of the user based on theproportions of the positive emotions, the negative emotions and theneutral emotions in all the current emotions of the user.

Referring to FIG. 5 , when the emotion proportion of the user isobtained, the current emotion state of the user is defined using apercentage average of the emotions. When the proportion of the positiveemotion of the user is greater than or equal to 75% (i.e. firstthreshold), the current emotion state of the user is a positive state.At this moment, the output result of the emotion analysis module 6 is 1,indicating that the value of the emotion state is 1. When the proportionof the negative emotion of the user is greater than or equal to 25%(i.e. second threshold), the current emotion state of the user is anegative state. At this moment, the output result of the emotionanalysis module 6 is -1, indicating that the value of the emotion stateis -1. When both the proportions of the positive emotion and thenegative emotion of the user are less than 25% and the proportion of theneutral emotion of the user is greater than or equal to 50% (i.e. thirdthreshold), the current emotion state of the user is a neutral state. Atthis moment, the output result of the emotion analysis module 6 is 0,indicating that the value of the emotion state is 0.

For example, in the above example, if the proportion of the negativeemotion is 33.3%, the current emotion state of the user is a negativestate, and the output result of the emotion analysis module 6 is -1.

Those skilled in the art will appreciate that the reason to dividenegative 25% from positive 75% is that the psychological perception isthat negative and positive feelings of subjective feelings of a personhave a weight difference of about 3:1, and three positive feelings areneeded to cancel one negative feeling. However, the above values havedifferent sizes according to different populations. For example, in theevaluation process for children, when the ratio of the positive emotionis greater than or equal to 80% (i.e. the threshold is increased), it isdetermined that the value of the emotion state is 1 since the positiveemotion of normal children is greater than that of adults. Similarly, inthe case of the evaluation of the elderly, when the ratio of thepositive emotion is greater than or equal to 70% (i.e. the threshold isreduced), it is determined that the value of the emotion state is 1.Different thresholds may also be set depending on gender or race.

S205: Acquire an emotion change state of the user based on an emotionproportion and a time change in each collection time period.

Referring to FIG. 6 , for example, taking 500 ms as a collection timeperiod, it can be seen from reference to step S201 that the userexpression collected in the collection time period needs to be comparedwith six basic emotions respectively, so as to obtain the proportions ofpositive emotions, negative emotions and neutral emotions in eachcollection time period.

The formulas show that the face collection device captures x expressionsof the user, and obtains a proportion (y1) of positive emotions and aproportion (y2) of negative emotions in each collection. Regressionanalysis (x and y1, x and y2) is carried out on the proportions ofpositive emotions and negative emotions, respectively, using a timefrequency to obtain normalized regression coefficients (b1, b2) asslopes and significance parameters (p1, p2) as significance referencesfor statistical significance. If p corresponds to max (|b1|, |b2|) isless than 0.05, |b1| and |b2| are compared. If |b1|>|b2| and b1>0, or|b1|<|b2| and b2<0, a positive change is made. If |b1|>|b2| and b1<0, or|b1|<|b2| and b2>0, a negative change is made. If p corresponds to max(|b1|, |b2|) is greater than 0.05, there is no change.

For example, the duration of a task is 120 seconds, and the facecollection device captures 240 expressions of user (x), and obtains aproportion (y1) of positive emotions and a proportion (y2) of negativeemotions in each of 240 collections. Regression analysis (x and y1, xand y2) is carried out on the proportions of positive emotions andnegative emotions, respectively, using a time frequency. If the positiveemotions are calculated: b1=0.25, p1=0.03, and negative emotions are:b2=-0.38, p1=0.01, p2<0.05, |b1|<|b2|, b2<0, the negative emotions aresignificantly reduced, a positive change is made, and the system outputresult is 1. If the positive emotions are calculated: b1=-0.28, p1=0.02,and negative emotions are: b2=-0.04, p1=0.64, p1<0.05, |b1|>|b2|, b1<0,the positive emotions are significantly reduced, a negative change ismade, and the system output result is -1. If the positive emotions arecalculated: b1=-0.12, p1=0.32, and negative emotions are: b2=0.04,p1=0.44, p1>0.05, neither the positive nor negative emotion issignificantly changed, there is no change, and the system output resultis 0.

It will be appreciated that a larger slope represents a greater increasein emotions over time. For example, positive emotions are the greatestincrease in slope, which means that the user feels happier whenperforming more tasks. A slope of the positive emotion and the negativeemotion changing with time throughout the full time history is acquiredbased on the emotion proportion situation of the user throughout thefull time history. Then, an emotion with the maximum slope change isobtained by comparison.

If the emotion with the maximum slope change is a descending negativeemotion or an ascending positive emotion and the slope change issignificantly not equal to 0, the emotion change state of the user is apositive change, and at this moment, the system output result is 1. Ifthe emotion with the maximum slope change is an ascending negativeemotion or a descending positive emotion and the slope change issignificantly not equal to 0, the emotion change state of the user is anegative change, and at this moment, the system output result is -1. Ifthe maximum slope change is not significantly distinguished from 0, theemotion change state of the user is no change, and at this moment, thesystem output result is 0.

It will be appreciated that the neutral emotion is not considered in theslope change because it is difficult to define from which part or wherethe change of the neutral emotion comes, it is meaningful to compare theneutral emotion rather than considering the meaning of the change of theneutral emotion departing from the positive and negative emotions.

S206: Acquire an emotion fluctuation state of the user based on theemotion proportion and the time change in each collection time period.

As shown in FIG. 7 , the emotion with the highest similarity proportionwhen recorded in each collection time period is defined as the currentprincipal emotion component, referring to steps S201-S202:

The principal emotion components of two adjacent collection time periodsare compared, and it is analyzed whether the principal emotion componentchanges.

If the principal emotion component changes, it is recorded as 1,otherwise, it is recorded as 0.

A principal emotion component change ratio in each task stage iscalculated according to the record results of whether the principalemotion component changes.

If the principal emotion component change ratio is less than 10% (i.e.fourth threshold), there is a normal fluctuation, and at this moment,the system output result is 1. If the principal emotion component changeratio is greater than or equal to 10% and less than or equal to 30%(i.e. fifth threshold), there is a slight fluctuation, and at thismoment, the system output result is 0. If the principal emotioncomponent change ratio is greater than 30%, there is a severefluctuation, and at this moment, the system output result is -1.

For example, the duration of a certain task is 75 seconds. When theprincipal emotion component of 56 seconds changes, the variability is56/75=74.7%, and the ratio of non-variability is 25.3%. Since 74.7% isgreater than 30%, it is determined that the emotion fluctuation of theuser is severe, and the system output result is -1.

It will be appreciated that in one embodiment of the present disclosure,10% and 30% are defined from the system evaluating the user. Thecollected face data is subjectively evaluated by tens of usersthereafter whether the emotion is in no fluctuation/slightfluctuation/severe fluctuation, and the current objective face emotionchange rate is calculated. It is concluded that the boundary between nofluctuation and slight fluctuation is the principal emotion componentchange ratio of 10%, and the boundary between slight fluctuation andsevere fluctuation is the principal emotion component change ratio of30%.

S207: Obtain the emotion comprehensive index of the user according to atleast two of the current emotion state, the emotion change state and theemotion fluctuation state.

As shown in FIG. 8 , the system output results of the current emotionstate, the emotion change state and the emotion fluctuation state areadded to obtain an emotion comprehensive index of the user, and theemotion situation of the user can be determined according to the emotioncomprehensive index of the user in accordance with the followingspecific determination criteria:

If the sum of the three output results is greater than 0, the emotioncomprehensive index is positive, and the system output emotioncomprehensive index is 1. If the three output results are less than 0,the emotion comprehensive index is negative, and the system outputemotion comprehensive index is -1. If the three output results are equalto 0, the emotion comprehensive index is neutral, and the system outputemotion comprehensive index is 0.

For example, the current emotion state of the user is a positive state,and the system output result is 1. The emotion change state of the useris a negative change state, and the system output result is -1. Theemotion fluctuation state of the user is a normal fluctuation, thesystem output result is 1, and then the emotion comprehensive index isequal to 1+(-1)+1=1>0. Therefore, the emotion comprehensive index of theuser is positive. At this moment, the emotion comprehensive index isobtained by integrating the current emotion state, the emotion changestate and the emotion fluctuation state. In another example, the currentemotion state of the user is a positive state, and the system outputresult is 1. The emotion change state of the user is a negative changestate, and the system output result is -1. Then the emotioncomprehensive index is equal to 1+(-1)=0. Therefore, the emotioncomprehensive index of the user is neutral. At this moment, the emotioncomprehensive index is obtained by integrating the current emotionstate, the emotion change state and the emotion fluctuation state. Inanother example, the emotion change state of the user is a negativechange state, and the system output result is -1. The emotionfluctuation state of the user is a severe fluctuation, the system outputresult is -1, and then the emotion comprehensive index is equal to(-1)+(-1)=-2<0. The emotion comprehensive index of the user is negative.At this moment, the emotion comprehensive index is obtained byintegrating the emotion change state and the emotion fluctuation state.In another example, the current emotion state of the user is a positivestate, and the system output result is 1. The emotion fluctuation stateof the user is a normal fluctuation, the system output result is 1, andthen the emotion comprehensive index is equal to 1+1=2>0. Therefore, theemotion comprehensive index of the user is positive. At this moment, theemotion comprehensive index is obtained by integrating the currentemotion state and the emotion fluctuation state.

It will be appreciated that in one embodiment of the present disclosure,the determination accuracy of the emotion comprehensive index can beimproved by using the current emotion state, the emotion change stateand the emotion fluctuation state as the determination basis, and inother embodiments, any two of the current emotion state, the emotionchange state and the emotion fluctuation state may be selected to becombined to determine the emotion comprehensive index.

In addition, in another embodiment, different weights may be given tothe current emotion state, the emotion change state and the emotionfluctuation state. Then, the emotion comprehensive index of the user maybe calculated by weighting, and the emotion comprehensive index of theuser may be calculated in a manner not limited to the above addition ofthe three output results.

S30: Acquire a human-computer interaction scheme of the user accordingto the evaluation result and the emotion comprehensive index.

Specifically, when the cognitive evaluation of the user is completed,the cognitive impairment level of the user is determined according tothe evaluation result, and the human-computer interaction task level ofthe user is jointly determined according to the emotion comprehensiveindex (i.e. emotion monitoring A) of the user in the evaluation process.

For example, after the user completes the evaluation (i.e. aftercompleting step S10), the system provides a comprehensive evaluationlevel of the brain ability of the user according to the norm standard ofeach task: good, mild and severe, and an emotion comprehensive index inthe evaluation process: positive, neutral and negative. After acquiringthe results of the two indexes, the human-computer interaction schemelevel is selected, and a higher level represents a more difficulthuman-computer interaction task.

The specific selection rule may be seen in Table 2:

TABLE 2 Determine human-computer interaction task level comprehensivelythrough emotion-brain ability Comprehensive evaluation level of brainability Good Mild Severe Emotion comprehensive index Positive High-orderHigh-order Medium-order Neutral High-order Medium-order Low-orderNegative Medium-order Low-order Low-order

It will be appreciated that the difficulty coefficient of thehuman-computer interaction scheme is increased in sequence from a loworder to a high order, and the number and types of human-computerinteraction tasks included in the human-computer interaction schemes ofdifferent orders are different. For example, the low-orderhuman-computer interaction scheme may include 3-5 simple human-computerinteraction tasks, the medium-order human-computer interaction schememay include 4-6 medium human-computer interaction tasks, and thehigh-order human-computer interaction scheme may include 5-8 difficulthuman-computer interaction tasks, whereby the most suitablehuman-computer interaction scheme can be selected according to thesituation of different users.

S40: Obtain an emotion comprehensive index after the human-computerinteraction according to the emotion monitoring of the user incompleting the human-computer interaction task, and obtain ahuman-computer interaction result of the user.

Specifically, after a corresponding equal-order human-computerinteraction scheme is generated for the user, the system will push aninitial human-computer interaction task, and when the user carries outthe human-computer interaction, the system will record the process ofthe human-computer interaction during the task completion so as tocollect human-computer interaction data, thereby acquiring thehuman-computer interaction result of the user. Meanwhile, in the processof completing the human-computer interaction task, the face expressionis collected by the face collection device, whereby the emotioncomprehensive index is analyzed (i.e. emotion monitoring B) by themethod described in step 20, so as to obtain the emotion comprehensiveindex of the user in the process of completing the human-computerinteraction task. It will be appreciated that both emotion monitoring Aand emotion monitoring B are the analysis of the emotion comprehensiveindex through step 20, except that the timing of the emotion monitoringis different. Emotion monitoring A is the emotion monitoring on the userwhen the user carries out the evaluation, and emotion monitoring B isthe emotion monitoring on the user when the user carries out thehuman-computer interaction task.

S50: Calculate and push the next human-computer interaction taskaccording to the human-computer interaction result and the emotioncomprehensive index after the human-computer interaction until all thetasks in the human-computer interaction scheme are completed.

Specifically, M (M is a positive integer, similarly hereinafter)human-computer interaction tasks are included in a human-computerinteraction scheme. Starting from an initial human-computer interactiontask (i.e. N=1), it is determined whether the number N (N is a positiveinteger, similarly hereinafter) of human-computer interaction tasks thathave been completed is equal to M. If N is equal to M, thehuman-computer interaction scheme ends. If N is less than M, the nexthuman-computer interaction task (i.e. N=2) is pushed through thehuman-computer interaction result of the previous human-computerinteraction task and the emotion comprehensive index of the user in theprevious human-computer interaction task. After the secondhuman-computer interaction scheme is completed, it is determined whetherN is equal to M again, and the operations are successively circulateduntil N=M, thereby completing the whole human-computer interactionscheme.

When calculating the next human-computer interaction task, if theemotion comprehensive index is negative, the task difficulty is reduced,and if the human-computer interaction result is good, the taskdifficulty is upgraded.

The specific pushing plan is shown with reference to Table 3:

TABLE 3 Emotion-human-computer interaction regulation Emotioncomprehensive index Positive 1 Normal 0 Negative -1 Human-computerinteraction result Good 1 Upgrade Upgrade Reduce pushing, updateModerate 0 Increase pushing Maintain Reduce pushing Poor -1 Increasepushing, degrade Degrade Degrade

In this embodiment, the human-computer interaction result is dividedinto good, moderate and poor, and the emotion comprehensive index isdivided into positive, normal and negative. Meanwhile, it will beappreciated that pushing is increased and reduced aiming athuman-computer interaction tasks of the same level, and the number ofhuman-computer interaction tasks is increased or reduced. Upgrading ordegrading represents the increase or reduction of the difficulty levelof the human-computer interaction task.

When the human-computer interaction result is good, task pushing issuccessively carried out according to upgrade, upgrade, and reducepushing and upgrade in the order of the emotion comprehensive indexbeing positive, normal and negative.

When the human-computer interaction result is moderate, task pushing issuccessively carried out according to increase pushing, maintain, andreduce pushing in the order of the emotion comprehensive index beingpositive, normal and negative.

When the human-computer interaction result is poor, task pushing issuccessively carried out according to increase pushing and degrade,degrade, and degrade in the order of the emotion comprehensive indexbeing positive, normal and negative.

In summary, the embodiments of the present disclosure provide acognitive disorder human-computer interaction method based on emotionmonitoring. Cognitive evaluation and human-computer interaction tasksare mainly included, and a variety of basic emotion information of auser in the process of executing tasks is monitored simultaneously inreal time, and an emotion comprehensive index of the user is obtained.Subsequent human-computer interaction tasks and states of the user areadjusted through the emotion comprehensive index, so as to achieve thepurpose of improving the cognitive disorder human-computer interactioneffect.

Second Embodiment

As shown in FIG. 9 , after step S40, the above-mentioned cognitivedisorder human-computer interaction method further includes thefollowing steps:

S40A: Carry out, in the process of human-computer interaction, a relaxedhuman-computer interaction between two adjacent human-computerinteraction tasks, and select a task type of the relaxed human-computerinteraction according to an emotion comprehensive index of the user inthe previous human-computer interaction task.

The specific selection rule is shown with reference to Table 4:

TABLE 4 Emotion comprehensive index Relaxed task selection rule Positiveindex High strength challenge class Negative index High relaxation andstress relief class Neutral index Random selection

When the emotion comprehensive index is positive, a human-computerinteraction task of a high strength challenge class is selected, whenthe emotion comprehensive index is negative, a human-computerinteraction task of a high relaxation and stress relief class isselected, and when the emotion comprehensive index is neutral, ahuman-computer interaction task is randomly selected from the foregoingtwo tasks. It will be appreciated that in this embodiment, the task ofthe high strength challenge class refers to a task with a higherdifficulty level and requiring the user to complete with own efforts,and the task of the high relaxation and stress relief class refers to atask with a lower difficulty level and being easily completed by theuser.

It will be appreciated that the existence of step S40A is specificallydetermined according to the actual situation. When a human-computerinteraction task is completed once, it is necessary to determine whetherthe whole human-computer interaction scheme is completed (i.e. whether Nis equal to M). If the whole human-computer interaction scheme is notcompleted (i.e. N is not equal to M), step S40A is executed to push therelaxed interaction task. If the whole human-computer interaction schemeis completed, it is not necessary to carry out the relaxed interactiontask. For example, a human-computer interaction scheme includes aplurality of human-computer interaction tasks (i.e. M>1), and then thewhole interaction process is: a mode of human-computerinteraction-relaxed interaction-human-computer interaction-relaxedinteraction. If a human-computer interaction scheme includes only onehuman-computer interaction task (i.e. M=1), there is no relaxedinteraction mode.

Meanwhile, in this embodiment, emotion monitoring (i.e. emotionmonitoring C) is also carried out when the user carries out relaxedinteraction. It will be appreciated that the steps of emotion monitoringduring relaxed interaction are the same as the steps of emotionmonitoring A and emotion monitoring B, except that the monitoring timingof emotion monitoring C is when the user carries out relaxedinteraction. However, emotion monitoring of relaxed interaction is onlyused to ensure that the user relaxes successfully and does not serve asa push basis for the next human-computer interaction task. The pushingof the next human-computer interaction task is based on thehuman-computer interaction result of the previous human-computerinteraction task and the emotion comprehensive index (i.e. emotionmonitoring B) during the previous human-computer interaction task.

Third Embodiment

Referring to FIG. 10 , this embodiment differs from the secondembodiment in that in this embodiment, emotion monitoring is carried outwhen a user carries out a relaxed interaction (i.e. emotion monitoringC). Meanwhile, an emotion comprehensive index obtained during therelaxed interaction is taken as a push basis for the next human-computerinteraction task.

Specifically, in this embodiment, after the previous human-computerinteraction task is completed, the user will enter a relaxed interactionmode. In the process of relaxed interaction, an emotion comprehensiveindex of the user in the process of relaxed interaction is obtained bycarrying out emotion monitoring (i.e. emotion monitoring C) on the user.Finally, the next human-computer interaction task is pushed incooperation with the emotion comprehensive index of the user in theprocess of relaxed interaction according to the human-computerinteraction result of the previous human-computer interaction task ofthe user, and this loop is continued until the whole human-computerinteraction scheme is completed.

Fourth Embodiment

As shown in FIG. 2 , this embodiment includes the following steps:

S10′: Obtain a cognitive ability level of a user.

S30′: Generate a human-computer interaction scheme according to thecognitive ability level.

S40′: Generate an N^(th) human-computer interaction task according tothe human-computer interaction scheme.

S50′: Carry out emotion monitoring on the performance of the user in theN^(th) human-computer interaction task.

S60′: Adjust the human-computer interaction scheme generated in S30′based on the result of emotion monitoring to generate a newhuman-computer interaction scheme, and return to S40′ of generating anN^(th) human-computer interaction task according to the newhuman-computer interaction scheme, whereby N=N+1 until all M tasks inthe new human-computer interaction scheme are completed.

The cognitive ability level in step S10′ may be obtained based on aconventional cognitive ability evaluation method, or using anear-infrared brain imaging map of the brain, etc.

The human-computer interaction scheme generated in S30′ is adjustedbased on the result of emotion monitoring in step S60′.

As shown in FIG. 11 , an embodiment of the present disclosure alsoprovides another cognitive disorder human-computer interaction systembased on emotion monitoring, including: an interaction unit, a storageunit and a processing unit. The system may be a server and a pluralityof communication terminals (for example, personal computers, tabletcomputers or mobile phones, etc.), and may also be a computer or amobile phone, etc.

The interaction unit communicates with the processing unit and isconfigured to receive information input by a user or output informationto the user. The interaction unit may be a keyboard and a display, andmay also be an input/output component of a smart phone or a tabletcomputer.

The storage unit is electrically connected to the processing unit forstoring an associated computer program for executing the human-computerinteraction method provided by the present disclosure. Therefore, dataassociated with the human-computer interaction and evaluation, etc. ofthe user may be retrieved repeatedly. The storage unit may be a harddisk in a computer, or a U disk, or a storage component of a smart phoneor a tablet computer.

The processing unit may be a central processing unit of a computer or aprocessor of a smart phone or tablet computer, and is configured to readthe computer program to execute the cognitive disorder human-computerinteraction method provided by the embodiments of the presentdisclosure.

The cognitive disorder human-computer interaction method and systembased on emotion monitoring provided in the present disclosure aredescribed in detail above. For a person of ordinary skill in the art,any obvious modifications made to the present disclosure withoutdeparting from the essence of the present disclosure will constitute aninfringement of patent rights of the present disclosure, andcorresponding legal liabilities will be born.

What is claimed is:
 1. A cognitive disorder human-computer interactionmethod based on emotion monitoring, comprising the following steps:obtaining a cognitive ability level of a user; generating ahuman-computer interaction scheme according to the cognitive abilitylevel; generating an N^(th) human-computer interaction task according tothe human-computer interaction scheme, N being a positive integer;carrying out emotion monitoring on the performance of the user in theN^(th) human-computer interaction task, and returning to the previousstep of generating an N^(th) human-computer interaction task accordingto the human-computer interaction scheme, whereby N=N+1 until all tasksin the human-computer interaction scheme are completed; and carryingout, in the process of human-computer interaction, a relaxedhuman-computer interaction between two adjacent human-computerinteraction tasks, and selecting a task type of the relaxedhuman-computer interaction according to an emotion comprehensive indexof the user in the previous human-computer interaction task, whereinwhen the emotion comprehensive index is positive, a human-computerinteraction task of a high strength challenge class is selected; whenthe emotion comprehensive index is negative, a human-computerinteraction task of a high relaxation and stress relief class isselected; and when the emotion comprehensive index is neutral, ahuman-computer interaction task is randomly selected.
 2. The cognitivedisorder human-computer interaction method according to claim 1, whereinwhen the user completes the relaxed human-computer interaction, the nexthuman-computer interaction task is pushed according to a human-computerinteraction result of the previous human-computer interaction task andthe emotion comprehensive index of the user after the relaxedhuman-computer interaction.
 3. The cognitive disorder human-computerinteraction method according to claim 1, wherein the human-computerinteraction result is divided into good, moderate and poor, and theemotion comprehensive index is divided into positive, normal andnegative; when the human-computer interaction result is good, taskpushing is successively carried out according to upgrade, upgrade, andreduce pushing and upgrade in the order of the emotion comprehensiveindex being positive, normal and negative; when the human-computerinteraction result is moderate, task pushing is successively carried outaccording to increase pushing, maintain, and reduce pushing in the orderof the emotion comprehensive index being positive, normal and negative;and when the human-computer interaction result is poor, task pushing issuccessively carried out according to increase pushing and degrade,degrade, and degrade in the order of the emotion comprehensive indexbeing positive, normal and negative.
 4. The cognitive disorderhuman-computer interaction method according to claim 1, whereinacquiring an emotion comprehensive index of a user specificallycomprises: acquiring expression information of the user in eachcollection time period; acquiring expression features in the expressioninformation, and comparing the expression features with an Asian facedatabase so as to obtain proportions of a positive emotion, a negativeemotion and a neutral emotion in the expression information, the emotionwith the maximum proportion being a current emotion of the user;determining proportions of positive emotions, negative emotions andneutral emotions in all the current emotions of the user in the wholeprocess of human-computer interaction; acquiring a current emotion stateof the user based on the proportions of the positive emotions, thenegative emotions and the neutral emotions in all the current emotionsof the user; acquiring an emotion change state of the user based on anemotion proportion and a time change in each collection time period;acquiring an emotion fluctuation state of the user based on the emotionproportion and the time change in each collection time period; andobtaining the emotion comprehensive index of the user according to atleast two of the current emotion state, the emotion change state and theemotion fluctuation state.
 5. The cognitive disorder human-computerinteraction method according to claim 4, wherein the acquiring a currentemotion state of the user based on the proportions of the positiveemotions, the negative emotions and the neutral emotions in all thecurrent emotions of the user specifically comprises: when the proportionof the positive emotion of the user is greater than or equal to a firstthreshold, determining the current emotion state of the user as apositive state; when the proportion of the negative emotion of the useris greater than or equal to a second threshold, determining the currentemotion state of the user as a negative state; and when both theproportions of the positive emotion and the negative emotion of the userare less than the second threshold and the proportion of the neutralemotion of the user is greater than or equal to a third threshold,determining the current emotion state of the user as a neutral state. 6.The cognitive disorder human-computer interaction method according toclaim 4, wherein the acquiring an emotion change state of the user basedon an emotion proportion and a time change in each collection timeperiod specifically comprises: taking the time change as an X value,taking the emotion proportion of the user in each collection time periodas a Y value, and performing regression analysis using the X value andthe Y value to obtain a slope; acquiring a slope of the positive emotionand the negative emotion changing with time throughout the full timehistory; obtaining an emotion with the maximum slope change bycomparison; if the emotion with the maximum slope change is a descendingnegative emotion or an ascending positive emotion and the slope changeis significantly not equal to 0, determining the emotion change state ofthe user as a positive change; if the emotion with the maximum slopechange is an ascending negative emotion or a descending positive emotionand the slope change is significantly not equal to 0, determining theemotion change state of the user as a negative change; and if themaximum slope change is not significantly distinguished from 0,determining the emotion change state of the user as no change.
 7. Thecognitive disorder human-computer interaction method according to claim4, wherein the acquiring an emotion fluctuation state of the user basedon the emotion proportion and the time change in each collection timeperiod specifically comprises: defining the emotion with the highestproportion in each collection time period as the current principalemotion component; comparing the principal emotion components of twoadjacent collection time periods, and analyzing whether the principalemotion component changes; if the principal emotion component changes,recording as 1, otherwise, recording as 0; calculating a principalemotion component change ratio in each task stage according to therecord results of whether the principal emotion component changes; andif the principal emotion component change ratio is less than a fourththreshold, determining as a normal fluctuation, if the principal emotioncomponent change ratio is greater than or equal to the fourth thresholdand less than or equal to a fifth threshold, determining as a slightfluctuation, and if the principal emotion component change ratio isgreater than the fifth threshold, determining as a severe fluctuation.8. The cognitive disorder human-computer interaction method according toclaim 1, wherein user information and user informed consent areacquired, the user informed consent comprising at least an evaluationcontent, a human-computer interaction content introduction, a userinformation collection range, and a user information usage range; and ifthe user agrees, cognitive evaluation is started and a face collectiondevice is initiated to start collecting user expression data, and if theuser disagrees, the evaluation is stopped.
 9. A cognitive disorderhuman-computer interaction system based on emotion monitoring,comprising: an interaction unit, configured to receive information inputby a user or output information to the user; a storage unit, configuredto store a computer program; and a processing unit, configured to readthe computer program to execute the cognitive disorder human-computerinteraction method according to claim 1.